架設 Airflow 步驟比較複雜，今天這篇先聚焦於 docker-compose 如何寫

決定 Airflow 的 Database

我的版本選定 2.2.4，查看官方文件建議資料庫是 Postgres、MySQL。如果沒有特別設定則會用預設的 SQLite，但只僅限於開發使用。

Set up a Database Backend — Airflow Documentation

而我這次選用 Postgres (因為我跟他比較熟悉XD)

Airflow Webserver

PORT

這是提供我們去瀏覽 Airflow 的後台，可以在以下區塊去確認你想要指定的 PORT

command: bash -c "export PYTHONPATH=$PYTHONPATH:/opt/airflow && airflow webserver -p 8080"

記得將 PORT 映射到本機，不然會看不到XDD

Executor Types

Executor — Airflow Documentation

官方強烈建議使用 LocalExecutor 這裡我就恭敬不如從命XD

Airflow Scheduler

這是 Airflow 排程，而與 webserver 功能不同，我特別將這兩個服務架設在不同容器。

基本上跟 webserver 要吃的參數一樣，都必續提供 Postgres 和儲存發票資料的 MongoDB

docker-compose.yml

最後整理如下

version: '3'

services:
    postgres:
        container_name: postgres
        image: postgres:14
        restart: always
        environment:
          - POSTGRES_USER=postgres
          - POSTGRES_PASSWORD=postgres
          - TZ=Asia/Taipei
        healthcheck:
          test: ["CMD-SHELL", "pg_isready -U postgres"]
          interval: 10s
          timeout: 5s
          retries: 5
        volumes:
          - postgres_data:/var/lib/postgresql/data
    airflow_webserver:
        container_name: airflow_webserver
        build: "airflow/."
        depends_on:
          - postgres
        environment:
          - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:postgres@postgres:5432/invoice
          - AIRFLOW__CORE__EXECUTOR=LocalExecutor
          - AIRFLOW__CORE__LOAD_EXAMPLES=False
          - AIRFLOW__WEBSERVER__SECRET_KEY=secret
          - POSTGRES_USER=postgres
          - POSTGRES_PASSWORD=postgres
          - MONGO_URI=mongodb://host.docker.internal:27017/invoice
        volumes:
          - ./airflow/dags:/opt/airflow/dags
          - ./airflow/logs:/opt/airflow/logs
          - ./airflow/:/opt/airflow
        ports:
          - "8080:8080"
        command: bash -c "export PYTHONPATH=$PYTHONPATH:/opt/airflow && airflow webserver -p 8080"
    airflow_scheduler:
        container_name: airflow_scheduler
        build: "airflow/."
        depends_on:
          - postgres
        environment:
          - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:postgres@postgres:5432/invoice
          - AIRFLOW__CORE__EXECUTOR=LocalExecutor
          - AIRFLOW__CORE__LOAD_EXAMPLES=False
          - AIRFLOW__WEBSERVER__SECRET_KEY=secret
          - POSTGRES_USER=postgres
          - POSTGRES_PASSWORD=postgres
          - MONGO_URI=mongodb://host.docker.internal:27017/invoice
        volumes:
          - ./airflow/dags:/opt/airflow/dags
          - ./airflow/logs:/opt/airflow/logs
          - ./airflow/:/opt/airflow
        command: bash -c "export PYTHONPATH=$PYTHONPATH:/opt/airflow && airflow scheduler"
volumes:
  postgres_data:

以上可以參考連結